Display artifact reduction

ABSTRACT

In one embodiment, a computing system may access an image to be displayed by a display. The system may determine one or more first characteristics associated with a content of the image. The one or more first characteristics may include a contrast level of the content of the image with respect to a background of the image. The system may determine a first display persistence time period for the display to display the image based on the one or more first characteristics associated with the content of the image. The system may configure the display to display the image using the first display persistence time period.

PRIORITY

This application is a continuation under 35 U.S.C. § 120 of U.S. PatentApplication No. 17/314,347, filed 7 May 2021, which is incorporatedherein by reference.

TECHNICAL FIELD

This disclosure generally relates to display technology, in particulardisplay artifact reduction.

BACKGROUND

Artificial reality is a form of reality that has been adjusted in somemanner before presentation to a user, which may include, e.g., a virtualreality (VR), an augmented reality (AR), a mixed reality (MR), a hybridreality, or some combination and/or derivatives thereof. Artificialreality content may include completely generated content or generatedcontent combined with captured content (e.g., real-world photographs).The artificial reality content may include video, audio, hapticfeedback, or some combination thereof, and any of which may be presentedin a single channel or in multiple channels (such as stereo video thatproduces a three-dimensional effect to the viewer). Artificial realitymay be associated with applications, products, accessories, services, orsome combination thereof, that are, e.g., used to create content in anartificial reality and/or used in (e.g., perform activities in) anartificial reality. The artificial reality system that provides theartificial reality content may be implemented on various platforms,including a head-mounted display (HMD) connected to a host computersystem, a standalone HMD, a mobile device or computing system, or anyother hardware platform capable of providing artificial reality contentto one or more viewers.

SUMMARY OF PARTICULAR EMBODIMENTS

Particular embodiments described herein relate to systems and methods ofdetermining an optimized display persistence for each image frame basedon image content characteristics (e.g., spatial frequency spectrum,background contrast), eye/head motion velocity of the user, or/andgazing position of the user, to reduce the motion blur and at the sametime avoid flicker. The likelihood of an image to have visible motionblur may depend on the spatial frequency spectrum of the image content.Human visual system is more sensitive to image content with higherspatial frequencies (e.g., sharp edges). Thus, an image with highspatial frequency spectrum may be more likely to have visible motionblur. On the other hand, image content with lower spatial frequencies(e.g., smooth gradient changes) are less likely to have visible motionblur. The system may compute the spatial frequency spectrum of an imageto be displayed and use a prediction model to predict the likelihood forthat image to have visible motion blur and calculate correspondingpersistence values to avoid such motion blur. To achieve this, thesystem may generate the prediction model based on spatial frequencyspectrum of image content. The system may compare the spatial frequencyspectrums of the “real world” images and the “simulated images” outputby the display to determine the difference of the power spectrums thatare within the visible range to human visual system. The visible rangeof human visual system may be described by the Window of Visibility(defined on temporal and spatial frequency domains) which can be used asa spatiotemporal filter during the modeling process. The predictionmodel may include a mapping model as represented by a lookup table thatcan be stored in memory and accessed at run time. The mapping model maymap the spatial frequency to corresponding persistence values. At runtime, the system may access the image to be displayed, compute thespatial frequency spectrum of that image, and compare the spatialfrequency characteristics to the stored data in the lookup table todetermine the appropriate display persistence for that image.Alternatively, instead of using a lookup table, the mapping model may berepresented by a transfer function which can be used by the system tocalculate, at the run time, the likelihood for that image to havevisible motion blur and the corresponding persistence value to avoid themotion blur. The transfer function may be pre-determined during themodeling process and may be used to map the spatial frequency spectrumof images to corresponding persistence values. The system may also usethe temporal frequency characteristics in temporal frequency domain todetermine the display persistence.

The user's eye/head motion velocity is another factor that affects thelikelihood of an image to have visible motion blur. In general, thefaster the user moves his eye/head, the more likely the image will havevisible motion blur. In particular embodiments, the HMD may use a headtracking system to track the user's eye/head motion (e.g., motionvelocity, motion direction). Similar to the spatial frequency spectrum,the system may generate a prediction model for predicting motion blurbased on the eye/head motion velocity and calculate the correspondingpersistence value. This prediction model may be built based on measureddata that correlates the eye/head motion velocity to the observed motionblur. The prediction model may include a mapping model as represented bya lookup table or a transfer function to determine the correspondingpersistence values. At the runtime, the system may access the look uptable or use the transfer function to determine the display persistencevalue based on the eye/head motion velocity.

In particular embodiments, the images displayed by HMD may havedifferent contrast levels with respect to the background (e.g., an ARobject with respect to real world background) which is another factorthat affects the likelihood for a displayed image to have visible motionblur. The system may use the background-foreground contrast (known asadditive contrast) to predict the likelihood for an image to havevisible motion blur. Similar to the factors of spatial frequencyspectrum and user eye/head motion velocity, the system may generate aprediction model that accounts for the background contrast. Theprediction model may include a mapping model as presented as a lookuptable or a transfer function. The system may use the prediction model todetermine the likelihood for each image frame to have visible motionblur and determine the corresponding display persistence value for thatimage using the mapping model.

In particular embodiments, the system may use respective predictionmodels for spatial frequency spectrum, user eye/head motion velocity, oradditive contrast to determine the respective display persistence valuesbased on respective factors. Then, the system may determine an optimizeddisplay persistence value based on two or more persistence valuesdetermined by the respective factors. For example, the system may choosethe lower persistence value to make sure the motion blur will bereduced. As another example, the system may use a weighted average asthe optimized persistence value to balance all factors considered. Inanother embodiment, the system may use a single prediction model whichcollectively takes in all factors of spatial frequency spectrum, usereye/head motion velocity, and additive contrast to determine theoptimized display persistence value. After the optimized persistencevalue is determined, the system may configure the display to display thecorresponding image using that optimized persistence value. The systemmay modulate persistence to a low level only when the image content ispredicted to likely to have visible motion blur. As a result, the systemcan reduce the motion blur artifact and at the same time have a balanceddisplay effect to avoid other artifacts such as flicker. Furthermore, byusing low persistence level only for the image that is predicted to havevisible motion blur, the system can reduce power consumption and improvedisplay panel lifetime.

The embodiments disclosed herein are only examples, and the scope ofthis disclosure is not limited to them. Particular embodiments mayinclude all, some, or none of the components, elements, features,functions, operations, or steps of the embodiments disclosed above.Embodiments according to the invention are in particular disclosed inthe attached claims directed to a method, a storage medium, a system anda computer program product, wherein any feature mentioned in one claimcategory, e.g. method, can be claimed in another claim category, e.g.system, as well. The dependencies or references back in the attachedclaims are chosen for formal reasons only. However, any subject matterresulting from a deliberate reference back to any previous claims (inparticular multiple dependencies) can be claimed as well, so that anycombination of claims and the features thereof are disclosed and can beclaimed regardless of the dependencies chosen in the attached claims.The subject-matter which can be claimed comprises not only thecombinations of features as set out in the attached claims but also anyother combination of features in the claims, wherein each featurementioned in the claims can be combined with any other feature orcombination of other features in the claims. Furthermore, any of theembodiments and features described or depicted herein can be claimed ina separate claim and/or in any combination with any embodiment orfeature described or depicted herein or with any of the features of theattached claims.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1A illustrates an example artificial reality system.

FIG. 1B illustrates an example augmented reality system.

FIG. 1C illustrates an example architecture of a display engine.

FIG. 1D illustrates an example graphic pipeline of the display enginefor generating display image data.

FIG. 2A illustrates an example scanning waveguide display.

FIG. 2B illustrates an example scanning operation of the scanningwaveguide display.

FIG. 3A illustrates an example 2D micro-LED waveguide display.

FIG. 3B illustrates an example waveguide configuration for the 2Dmicro-LED waveguide display.

FIG. 4 illustrates an example frame updating scheme using periodicaldisplay-on time.

FIG. 5 illustrates an example Window of Visibility for human visualsystem.

FIG. 6A illustrates an example process for using a prediction model topredict the likelihood for an image to have artifacts and to determinethe corresponding persistence value.

FIG. 6B illustrates an example mapping model which maps the inputfactors to corresponding persistence values.

FIGS. 7A-7B illustrate example images that have content with differentspatial frequencies and allow the display to use different displaypersistence values.

FIG. 8 illustrates an example process for determining an optimizedpersistence value based on multiple factors.

FIG. 9 illustrates an example method for adaptively determining displaypersistence time based on image content.

FIG. 10 illustrates an example computer system.

DESCRIPTION OF EXAMPLE EMBODIMENTS

FIG. 1A illustrates an example artificial reality system 100A. Inparticular embodiments, the artificial reality system 100 may comprise aheadset 104, a controller 106, and a computing system 108. A user 102may wear the headset 104 that may display visual artificial realitycontent to the user 102. The headset 104 may include an audio devicethat may provide audio artificial reality content to the user 102. Theheadset 104 may include one or more cameras which can capture images andvideos of environments. The headset 104 may include an eye trackingsystem to determine the vergence distance of the user 102. The headset104 may be referred as a head-mounted display (HDM). The controller 106may comprise a trackpad and one or more buttons. The controller 106 mayreceive inputs from the user 102 and relay the inputs to the computingsystem 108. The controller 206 may also provide haptic feedback to theuser 102. The computing system 108 may be connected to the headset 104and the controller 106 through cables or wireless connections. Thecomputing system 108 may control the headset 104 and the controller 106to provide the artificial reality content to and receive inputs from theuser 102. The computing system 108 may be a standalone host computersystem, an on-board computer system integrated with the headset 104, amobile device, or any other hardware platform capable of providingartificial reality content to and receiving inputs from the user 102.

FIG. 1B illustrates an example augmented reality system 100B. Theaugmented reality system 100B may include a head-mounted display (HMD)110 (e.g., glasses) comprising a frame 112, one or more displays 114,and a computing system 120. The displays 114 may be transparent ortranslucent allowing a user wearing the HMD 110 to look through thedisplays 114 to see the real world and displaying visual artificialreality content to the user at the same time. The HMD 110 may include anaudio device that may provide audio artificial reality content to users.The HMD 110 may include one or more cameras which can capture images andvideos of environments. The HMD 110 may include an eye tracking systemto track the vergence movement of the user wearing the HMD 110. Theaugmented reality system 100B may further include a controllercomprising a trackpad and one or more buttons. The controller mayreceive inputs from users and relay the inputs to the computing system120. The controller may also provide haptic feedback to users. Thecomputing system 120 may be connected to the HMD 110 and the controllerthrough cables or wireless connections. The computing system 120 maycontrol the HMD 110 and the controller to provide the augmented realitycontent to and receive inputs from users. The computing system 120 maybe a standalone host computer system, an on-board computer systemintegrated with the HMD 110, a mobile device, or any other hardwareplatform capable of providing artificial reality content to andreceiving inputs from users.

FIG. 1C illustrates an example architecture 100C of a display engine130. In particular embodiments, the processes and methods as describedin this disclosure may be embodied or implemented within a displayengine 130 (e.g., in the display block 135). The display engine 130 mayinclude, for example, but is not limited to, a texture memory 132, atransform block 133, a pixel block 134, a display block 135, input databus 131, output data bus 142, etc. In particular embodiments, thedisplay engine 130 may include one or more graphic pipelines forgenerating images to be rendered on the display. For example, thedisplay engine may use the graphic pipeline(s) to generate a series ofsubframe images based on a mainframe image and a viewpoint or view angleof the user as measured by one or more eye tracking sensors. Themainframe image may be generated or/and loaded in to the system at amainframe rate of 30-90 Hz and the subframe rate may be generated at asubframe rate of 1-2 kHz. In particular embodiments, the display engine130 may include two graphic pipelines for the user's left and righteyes. One of the graphic pipelines may include or may be implemented onthe texture memory 132, the transform block 133, the pixel block 134,the display block 135, etc. The display engine 130 may include anotherset of transform block, pixel block, and display block for the othergraphic pipeline. The graphic pipeline(s) may be controlled by acontroller or control block (not shown) of the display engine 130. Inparticular embodiments, the texture memory 132 may be included withinthe control block or may be a memory unit external to the control blockbut local to the display engine 130. One or more of the components ofthe display engine 130 may be configured to communicate via a high-speedbus, shared memory, or any other suitable methods. This communicationmay include transmission of data as well as control signals, interruptsor/and other instructions. For example, the texture memory 132 may beconfigured to receive image data through the input data bus 211. Asanother example, the display block 135 may send the pixel values to thedisplay system 140 through the output data bus 142. In particularembodiments, the display system 140 may include three color channels(e.g., 114A, 114B, 114C) with respective display driver ICs (DDIs) of142A, 142B, and 143B. In particular embodiments, the display system 140may include, for example, but is not limited to, light-emitting diode(LED) displays, organic light-emitting diode (OLED) displays, activematrix organic light-emitting diode (AMLED) displays, liquid crystaldisplay (LCD), micro light-emitting diode (μLED) display,electroluminescent displays (ELDs), or any suitable displays.

In particular embodiments, the display engine 130 may include acontroller block (not shown). The control block may receive data andcontrol packages such as position data and surface information fromcontrollers external to the display engine 130 though one or more databuses. For example, the control block may receive input stream data froma body wearable computing system. The input data stream may include aseries of mainframe images generated at a mainframe rate of 30-90 Hz.The input stream data including the mainframe images may be converted tothe required format and stored into the texture memory 132. Inparticular embodiments, the control block may receive input from thebody wearable computing system and initialize the graphic pipelines inthe display engine to prepare and finalize the image data for renderingon the display. The data and control packets may include informationrelated to, for example, one or more surfaces including texel data,position data, and additional rendering instructions. The control blockmay distribute data as needed to one or more other blocks of the displayengine 130. The control block may initiate the graphic pipelines forprocessing one or more frames to be displayed. In particularembodiments, the graphic pipelines for the two eye display systems mayeach include a control block or share the same control block.

In particular embodiments, the transform block 133 may determine initialvisibility information for surfaces to be displayed in the artificialreality scene. In general, the transform block 133 may cast rays frompixel locations on the screen and produce filter commands (e.g.,filtering based on bilinear or other types of interpolation techniques)to send to the pixel block 134. The transform block 133 may perform raycasting from the current viewpoint of the user (e.g., determined usingthe headset's inertial measurement units, eye tracking sensors, and/orany suitable tracking/localization algorithms, such as simultaneouslocalization and mapping (SLAM)) into the artificial scene wheresurfaces are positioned and may produce tile/surface pairs 144 to sendto the pixel block 134. In particular embodiments, the transform block133 may include a four-stage pipeline as follows. A ray caster may issueray bundles corresponding to arrays of one or more aligned pixels,referred to as tiles (e.g., each tile may include 16×16 aligned pixels).The ray bundles may be warped, before entering the artificial realityscene, according to one or more distortion meshes. The distortion meshesmay be configured to correct geometric distortion effects stemming from,at least, the eye display systems the headset system. The transformblock 133 may determine whether each ray bundle intersects with surfacesin the scene by comparing a bounding box of each tile to bounding boxesfor the surfaces. If a ray bundle does not intersect with an object, itmay be discarded. After the tile-surface intersections are detected, thecorresponding tile/surface pairs may be passed to the pixel block 134.

In particular embodiments, the pixel block 134 may determine colorvalues or grayscale values for the pixels based on the tile-surfacepairs. The color values for each pixel may be sampled from the texeldata of surfaces received and stored in texture memory 132. The pixelblock 134 may receive tile-surface pairs from the transform block 133and may schedule bilinear filtering using one or more filer blocks. Foreach tile-surface pair, the pixel block 134 may sample color informationfor the pixels within the tile using color values corresponding to wherethe projected tile intersects the surface. The pixel block 134 maydetermine pixel values based on the retrieved texels (e.g., usingbilinear interpolation). In particular embodiments, the pixel block 134may process the red, green, and blue color components separately foreach pixel. In particular embodiments, the display may include two pixelblocks for the two eye display systems. The two pixel blocks of the twoeye display systems may work independently and in parallel with eachother. The pixel block 134 may then output its color determinations(e.g., pixels 138) to the display block 135. In particular embodiments,the pixel block 134 may composite two or more surfaces into one surfaceto when the two or more surfaces have overlapping areas. A composedsurface may need less computational resources (e.g., computationalunits, memory, power, etc.) for the resampling process.

In particular embodiments, the display block 135 may receive pixel colorvalues from the pixel block 134, covert the format of the data to bemore suitable for the scanline output of the display, apply one or morebrightness corrections to the pixel color values, and prepare the pixelcolor values for output to the display. In particular embodiments, thedisplay block 135 may each include a row buffer and may process andstore the pixel data received from the pixel block 134. The pixel datamay be organized in quads (e.g., 2×2 pixels per quad) and tiles (e.g.,16×16 pixels per tile). The display block 135 may convert tile-orderpixel color values generated by the pixel block 134 into scanline orrow-order data, which may be required by the physical displays. Thebrightness corrections may include any required brightness correction,gamma mapping, and dithering. The display block 135 may output thecorrected pixel color values directly to the driver of the physicaldisplay (e.g., pupil display) or may output the pixel values to a blockexternal to the display engine 130 in a variety of formats. For example,the eye display systems of the headset system may include additionalhardware or software to further customize backend color processing, tosupport a wider interface to the display, or to optimize display speedor fidelity.

In particular embodiments, the dithering methods and processes (e.g.,spatial dithering method, temporal dithering methods, andspatio-temporal methods) as described in this disclosure may be embodiedor implemented in the display block 135 of the display engine 130. Inparticular embodiments, the display block 135 may include a model-baseddithering algorithm or a dithering model for each color channel and sendthe dithered results of the respective color channels to the respectivedisplay driver ICs (DDIs) (e.g., 142A, 142B, 142C) of display system140. In particular embodiments, before sending the pixel values to therespective display driver ICs (e.g., 142A, 142B, 142C), the displayblock 135 may further include one or more algorithms for correcting, forexample, pixel non-uniformity, LED non-ideality, waveguidenon-uniformity, display defects (e.g., dead pixels), displaydegradation, etc. U.S. patent application Ser. No. 16/998,860, entitled“Display Degradation Compensation,” first named inventor “EdwardBuckley,” filed on 20 Aug. 2020, which discloses example systems,methods, and processes for display degradation compensation, isincorporated herein by reference.

In particular embodiments, graphics applications (e.g., games, maps,content-providing apps, etc.) may build a scene graph, which is usedtogether with a given view position and point in time to generateprimitives to render on a GPU or display engine. The scene graph maydefine the logical and/or spatial relationship between objects in thescene. In particular embodiments, the display engine 130 may alsogenerate and store a scene graph that is a simplified form of the fullapplication scene graph. The simplified scene graph may be used tospecify the logical and/or spatial relationships between surfaces (e.g.,the primitives rendered by the display engine 130, such asquadrilaterals or contours, defined in 3D space, that have correspondingtextures generated based on the mainframe rendered by the application).Storing a scene graph allows the display engine 130 to render the sceneto multiple display frames and to adjust each element in the scene graphfor the current viewpoint (e.g., head position), the current objectpositions (e.g., they could be moving relative to each other) and otherfactors that change per display frame. In addition, based on the scenegraph, the display engine 130 may also adjust for the geometric andcolor distortion introduced by the display subsystem and then compositethe objects together to generate a frame. Storing a scene graph allowsthe display engine 130 to approximate the result of doing a full renderat the desired high frame rate, while actually running the GPU ordisplay engine 130 at a significantly lower rate.

FIG. 1D illustrates an example graphic pipeline 100D of the displayengine 130 for generating display image data. In particular embodiments,the graphic pipeline 100D may include a visibility step 152, where thedisplay engine 130 may determine the visibility of one or more surfacesreceived from the body wearable computing system. The visibility step152 may be performed by the transform block (e.g., 2133 in FIG. 1C) ofthe display engine 130. The display engine 130 may receive (e.g., by acontrol block or a controller) input data 151 from the body-wearablecomputing system. The input data 151 may include one or more surfaces,texel data, position data, RGB data, and rendering instructions from thebody wearable computing system. The input data 151 may include mainframeimages with 30-90 frames per second (FPS). The main frame image may havecolor depth of, for example, 24 bits per pixel. The display engine 130may process and save the received input data 151 in the texel memory132. The received data may be passed to the transform block 133 whichmay determine the visibility information for surfaces to be displayed.The transform block 133 may cast rays for pixel locations on the screenand produce filter commands (e.g., filtering based on bilinear or othertypes of interpolation techniques) to send to the pixel block 134. Thetransform block 133 may perform ray casting from the current viewpointof the user (e.g., determined using the headset's inertial measurementunits, eye trackers, and/or any suitable tracking/localizationalgorithms, such as simultaneous localization and mapping (SLAM)) intothe artificial scene where surfaces are positioned and producesurface-tile pairs to send to the pixel block 134.

In particular embodiments, the graphic pipeline 100D may include aresampling step 153, where the display engine 130 may determine thecolor values from the tile-surfaces pairs to produce pixel color values.The resampling step 153 may be performed by the pixel block 134 in FIG.1C) of the display engine 130. The pixel block 134 may receivetile-surface pairs from the transform block 133 and may schedulebilinear filtering. For each tile-surface pair, the pixel block 134 maysample color information for the pixels within the tile using colorvalues corresponding to where the projected tile intersects the surface.The pixel block 134 may determine pixel values based on the retrievedtexels (e.g., using bilinear interpolation) and output the determinedpixel values to the respective display block 135.

In particular embodiments, the graphic pipeline 100D may include a bendstep 154, a correction and dithering step 155, a serialization step 156,etc. In particular embodiments, the bend step, correction and ditheringstep, and serialization steps of 154, 155, and 156 may be performed bythe display block (e.g., 135 in FIG. 1C) of the display engine 130. Thedisplay engine 130 may blend the display content for display contentrendering, apply one or more brightness corrections to the pixel colorvalues, perform one or more dithering algorithms for dithering thequantization errors both spatially and temporally, serialize the pixelvalues for scanline output for the physical display, and generate thedisplay data 159 suitable for the display system 140. The display engine130 may send the display data 159 to the display system 140. Inparticular embodiments, the display system 140 may include three displaydriver ICs (e.g., 142A, 142B, 142C) for the pixels of the three colorchannels of RGB (e.g., 144A, 144B, 144C).

FIG. 2A illustrates an example scanning waveguide display 200A. Inparticular embodiments, the head-mounted display (HMD) of the AR/VRsystem may include a near eye display (NED) which may be a scanningwaveguide display 200A. The scanning waveguide display 200A may includea light source assembly 210, an output waveguide 204, a controller 216,etc. The scanning waveguide display 200A may provide images for botheyes or for a single eye. For purposes of illustration, FIG. 3A showsthe scanning waveguide display 200A associated with a single eye 202.Another scanning waveguide display (not shown) may provide image lightto the other eye of the user and the two scanning waveguide displays mayshare one or more components or may be separated. The light sourceassembly 210 may include a light source 212 and an optics system 214.The light source 212 may include an optical component that couldgenerate image light using an array of light emitters. The light source212 may generate image light including, for example, but not limited to,red image light, blue image light, green image light, infra-red imagelight, etc. The optics system 214 may perform a number of opticalprocesses or operations on the image light generated by the light source212. The optical processes or operations performed by the optics systems214 may include, for example, but are not limited to, light focusing,light combining, light conditioning, scanning, etc.

In particular embodiments, the optics system 214 may include a lightcombining assembly, a light conditioning assembly, a scanning mirrorassembly, etc. The light source assembly 210 may generate and output animage light 219 to a coupling element 218 of the output waveguide 204.The output waveguide 204 may be an optical waveguide that could outputimage light to the user eye 202. The output waveguide 204 may receivethe image light 219 at one or more coupling elements 218 and guide thereceived image light to one or more decoupling elements 206. Thecoupling element 218 may be, for example, but is not limited to, adiffraction grating, a holographic grating, any other suitable elementsthat can couple the image light 219 into the output waveguide 204, or acombination thereof. As an example and not by way of limitation, if thecoupling element 350 is a diffraction grating, the pitch of thediffraction grating may be chosen to allow the total internal reflectionto occur and the image light 219 to propagate internally toward thedecoupling element 206. The pitch of the diffraction grating may be inthe range of 300 nm to 600 nm. The decoupling element 206 may decouplethe total internally reflected image light from the output waveguide204. The decoupling element 206 may be, for example, but is not limitedto, a diffraction grating, a holographic grating, any other suitableelement that can decouple image light out of the output waveguide 204,or a combination thereof. As an example and not by way of limitation, ifthe decoupling element 206 is a diffraction grating, the pitch of thediffraction grating may be chosen to cause incident image light to exitthe output waveguide 204. The orientation and position of the imagelight exiting from the output waveguide 204 may be controlled bychanging the orientation and position of the image light 219 enteringthe coupling element 218. The pitch of the diffraction grating may be inthe range of 300 nm to 600 nm.

In particular embodiments, the output waveguide 204 may be composed ofone or more materials that can facilitate total internal reflection ofthe image light 219. The output waveguide 204 may be composed of one ormore materials including, for example, but not limited to, silicon,plastic, glass, polymers, or some combination thereof. The outputwaveguide 204 may have a relatively small form factor. As an example andnot by way of limitation, the output waveguide 204 may be approximately50 mm wide along X-dimension, 30 mm long along Y-dimension and 0.5-1 mmthick along Z-dimension. The controller 216 may control the scanningoperations of the light source assembly 210. The controller 216 maydetermine scanning instructions for the light source assembly 210 basedat least on the one or more display instructions for rendering one ormore images. The display instructions may include an image file (e.g.,bitmap) and may be received from, for example, a console or computer ofthe AR/VR system. Scanning instructions may be used by the light sourceassembly 210 to generate image light 219. The scanning instructions mayinclude, for example, but are not limited to, an image light source type(e.g., monochromatic source, polychromatic source), a scanning rate, ascanning apparatus orientation, one or more illumination parameters, orsome combination thereof. The controller 216 may include a combinationof hardware, software, firmware, or any suitable components supportingthe functionality of the controller 216.

FIG. 2B illustrates an example scanning operation of a scanningwaveguide display 200B. The light source 220 may include an array oflight emitters 222 (as represented by the dots in inset) with multiplerows and columns. The light 223 emitted by the light source 220 mayinclude a set of collimated beams of light emitted by each column oflight emitters 222. Before reaching the mirror 224, the light 223 may beconditioned by different optical devices such as the conditioningassembly (not shown). The mirror 224 may reflect and project the light223 from the light source 220 to the image field 227 by rotating aboutan axis 225 during scanning operations. The mirror 224 may be amicroelectromechanical system (MEMS) mirror or any other suitablemirror. As the mirror 224 rotates about the axis 225, the light 223 maybe projected to a different part of the image field 227, as illustratedby the reflected part of the light 226A in solid lines and the reflectedpart of the light 226B in dash lines.

In particular embodiments, the image field 227 may receive the light226A-B as the mirror 224 rotates about the axis 225 to project the light226A-B in different directions. For example, the image field 227 maycorrespond to a portion of the coupling element 218 or a portion of thedecoupling element 206 in FIG. 2A. In particular embodiments, the imagefield 227 may include a surface of the coupling element 206. The imageformed on the image field 227 may be magnified as light travels throughthe output waveguide 220. In particular embodiments, the image field 227may not include an actual physical structure but include an area towhich the image light is projected to form the images. The image field227 may also be referred to as a scan field. When the light 223 isprojected to an area of the image field 227, the area of the image field227 may be illuminated by the light 223. The image field 227 may includea matrix of pixel locations 229 (represented by the blocks in inset 228)with multiple rows and columns. The pixel location 229 may be spatiallydefined in the area of the image field 227 with a pixel locationcorresponding to a single pixel. In particular embodiments, the pixellocations 229 (or the pixels) in the image field 227 may not includeindividual physical pixel elements. Instead, the pixel locations 229 maybe spatial areas that are defined within the image field 227 and dividethe image field 227 into pixels. The sizes and locations of the pixellocations 229 may depend on the projection of the light 223 from thelight source 220. For example, at a given rotation angle of the mirror224, light beams emitted from the light source 220 may fall on an areaof the image field 227. As such, the sizes and locations of pixellocations 229 of the image field 227 may be defined based on thelocation of each projected light beam. In particular embodiments, apixel location 229 may be subdivided spatially into subpixels (notshown). For example, a pixel location 229 may include a red subpixel, agreen subpixel, and a blue subpixel. The red, green and blue subpixelsmay correspond to respective locations at which one or more red, greenand blue light beams are projected. In this case, the color of a pixelmay be based on the temporal and/or spatial average of the pixel'ssubpixels.

In particular embodiments, the light emitters 222 may illuminate aportion of the image field 227 (e.g., a particular subset of multiplepixel locations 229 on the image field 227) with a particular rotationangle of the mirror 224. In particular embodiment, the light emitters222 may be arranged and spaced such that a light beam from each of thelight emitters 222 is projected on a corresponding pixel location 229.In particular embodiments, the light emitters 222 may include a numberof light-emitting elements (e.g., micro-LEDs) to allow the light beamsfrom a subset of the light emitters 222 to be projected to a same pixellocation 229. In other words, a subset of multiple light emitters 222may collectively illuminate a single pixel location 229 at a time. As anexample and not by way of limitation, a group of light emitter includingeight light-emitting elements may be arranged in a line to illuminate asingle pixel location 229 with the mirror 224 at a given orientationangle.

In particular embodiments, the number of rows and columns of lightemitters 222 of the light source 220 may or may not be the same as thenumber of rows and columns of the pixel locations 229 in the image field227. In particular embodiments, the number of light emitters 222 in arow may be equal to the number of pixel locations 229 in a row of theimage field 227 while the light emitters 222 may have fewer columns thanthe number of pixel locations 229 of the image field 227. In particularembodiments, the light source 220 may have the same number of columns oflight emitters 222 as the number of columns of pixel locations 229 inthe image field 227 but fewer rows. As an example and not by way oflimitation, the light source 220 may have about 1280 columns of lightemitters 222 which may be the same as the number of columns of pixellocations 229 of the image field 227, but only a handful rows of lightemitters 222. The light source 220 may have a first length L1 measuredfrom the first row to the last row of light emitters 222. The imagefield 530 may have a second length L2, measured from the first row(e.g., Row 1) to the last row (e.g., Row P) of the image field 227. TheL2 may be greater than L1 (e.g., L2 is 50 to 10,000 times greater thanL1).

In particular embodiments, the number of rows of pixel locations 229 maybe larger than the number of rows of light emitters 222. The displaydevice 200B may use the mirror 224 to project the light 223 to differentrows of pixels at different time. As the mirror 520 rotates and thelight 223 scans through the image field 227, an image may be formed onthe image field 227. In some embodiments, the light source 220 may alsohas a smaller number of columns than the image field 227. The mirror 224may rotate in two dimensions to fill the image field 227 with light, forexample, using a raster-type scanning process to scan down the rows thenmoving to new columns in the image field 227. A complete cycle ofrotation of the mirror 224 may be referred to as a scanning period whichmay be a predetermined cycle time during which the entire image field227 is completely scanned. The scanning of the image field 227 may bedetermined and controlled by the mirror 224 with the light generation ofthe display device 200B being synchronized with the rotation of themirror 224. As an example and not by way of limitation, the mirror 224may start at an initial position projecting light to Row 1 of the imagefield 227, and rotate to the last position that projects light to Row Pof the image field 227, and then rotate back to the initial positionduring one scanning period. An image (e.g., a frame) may be formed onthe image field 227 per scanning period. The frame rate of the displaydevice 200B may correspond to the number of scanning periods in asecond. As the mirror 224 rotates, the light may scan through the imagefield to form images. The actual color value and light intensity orbrightness of a given pixel location 229 may be a temporal sum of thecolor various light beams illuminating the pixel location during thescanning period. After completing a scanning period, the mirror 224 mayrevert back to the initial position to project light to the first fewrows of the image field 227 with a new set of driving signals being fedto the light emitters 222. The same process may be repeated as themirror 224 rotates in cycles to allow different frames of images to beformed in the scanning field 227.

FIG. 3A illustrates an example 2D micro-LED waveguide display 300A. Inparticular embodiments, the display 300A may include an elongatewaveguide configuration 302 that may be wide or long enough to projectimages to both eyes of a user. The waveguide configuration 302 mayinclude a decoupling area 304 covering both eyes of the user. In orderto provide images to both eyes of the user through the waveguideconfiguration 302, multiple coupling areas 306A-B may be provided in atop surface of the waveguide configuration 302. The coupling areas 306Aand 306B may include multiple coupling elements to receive image lightfrom light emitter array sets 308A and 308B, respectively. Each of theemitter array sets 308A-B may include a number of monochromatic emitterarrays including, for example, but not limited to, a red emitter array,a green emitter array, and a blue emitter array. In particularembodiments, the emitter array sets 308A-B may further include a whiteemitter array or an emitter array emitting other colors or anycombination of any multiple colors. In particular embodiments, thewaveguide configuration 302 may have the emitter array sets 308A and308B covering approximately identical portions of the decoupling area304 as divided by the divider line 309A. In particular embodiments, theemitter array sets 308A and 308B may provide images to the waveguide ofthe waveguide configuration 302 asymmetrically as divided by the dividerline 309B. For example, the emitter array set 308A may provide image tomore than half of the decoupling area 304. In particular embodiments,the emitter array sets 308A and 308B may be arranged at opposite sides(e.g., 180° apart) of the waveguide configuration 302 as shown in FIG.3B. In other embodiments, the emitter array sets 308A and 308B may bearranged at any suitable angles. The waveguide configuration 302 may beplanar or may have a curved cross-sectional shape to better fit to theface/head of a user.

FIG. 3B illustrates an example waveguide configuration 300B for the 2Dmicro-LED waveguide display. In particular embodiments, the waveguideconfiguration 300B may include a projector device 350 coupled to awaveguide 342. The projector device 320 may include a number of lightemitters 352 (e.g., monochromatic emitters) secured to a supportstructure 354 (e.g., a printed circuit board or other suitable supportstructure). The waveguide 342 may be separated from the projector device350 by an air gap having a distance of D1 (e.g., approximately 50 μm toapproximately 500 μm). The monochromatic images projected by theprojector device 350 may pass through the air gap toward the waveguide342. The waveguide 342 may be formed from a glass or plastic material.The waveguide 342 may include a coupling area 330 including a number ofcoupling elements 334A-C for receiving the emitted light from theprojector device 350. The waveguide 342 may include a decoupling areawith a number of decoupling elements 336A on the top surface 318A and anumber of decoupling elements 336B on the bottom surface 318B. The areawithin the waveguide 342 in between the decoupling elements 336A and336B may be referred as a propagation area 310, in which image lightreceived from the projector device 350 and coupled into the waveguide342 by the coupling element 334 may propagate laterally within thewaveguide 342.

The coupling area 330 may include coupling elements (e.g., 334A, 334B,334C) configured and dimensioned to couple light of predeterminedwavelengths (e.g., red, green, blue). When a white light emitter arrayis included in the projector device 350, the portion of the white lightthat falls in the predetermined wavelengths may be coupled by each ofthe coupling elements 334A-C. In particular embodiments, the couplingelements 334A-B may be gratings (e.g., Bragg gratings) dimensioned tocouple a predetermined wavelength of light. In particular embodiments,the gratings of each coupling element may exhibit a separation distancebetween gratings associated with the predetermined wavelength of lightand each coupling element may have different grating separationdistances. Accordingly, each coupling element (e.g., 334A-C) may couplea limited portion of the white light from the white light emitter arrayof the projector device 350 if white light emitter array is included inthe projector device 350. In particular embodiments, each couplingelement (e.g., 334A-C) may have the same grating separation distance. Inparticular embodiments, the coupling elements 334A-C may be or include amultiplexed coupler.

As illustrated in FIG. 3B, a red image 320A, a blue image 320B, and agreen image 320C may be coupled by the coupling elements 334A, 334B,334C, respectively, into the propagation area 310 and may begin totraverse laterally within the waveguide 342. A portion of the light maybe projected out of the waveguide 342 after the light contacts thedecoupling element 336A for one-dimensional pupil replication, and afterthe light contacts both the decoupling elements 336A and 336B fortwo-dimensional pupil replication. In two-dimensional pupil replication,the light may be projected out of the waveguide 342 at locations wherethe pattern of the decoupling element 336A intersects the pattern of thedecoupling element 336B. The portion of the light that is not projectedout of the waveguide 342 by the decoupling element 336A may be reflectedoff the decoupling element 336B. The decoupling element 336B may reflectall incident light back toward the decoupling element 336A. Accordingly,the waveguide 342 may combine the red image 320A, the blue image 320B,and the green image 320C into a polychromatic image instance which maybe referred as a pupil replication 322. The polychromatic pupilreplication 322 may be projected to the user's eyes which may interpretthe pupil replication 322 as a full color image (e.g., an imageincluding colors addition to red, green, and blue). The waveguide 342may produce tens or hundreds of pupil replication 322 or may produce asingle replication 322.

In particular embodiments, the AR/VR system may use scanning waveguidedisplays or 2D micro-LED displays for displaying AR/VR content to users.In order to miniaturize the AR/VR system, the display system may need tominiaturize the space for pixel circuits and may have limited number ofavailable bits for the display. The number of available bits in adisplay may limit the display's color depth or gray scale level, andconsequently limit the quality of the displayed images. Furthermore, thewaveguide displays used for AR/VR systems may have nonuniformity problemcross all display pixels. The compensation operations for pixelnonuniformity may result in loss on image grayscale and further reducethe quality of the displayed images. For example, a waveguide displaywith 8-bit pixels (i.e., 256 gray level) may equivalently have 6-bitpixels (i.e., 64 gray level) after compensation of the nonuniformity(e.g., 8:1 waveguide nonuniformity, 0.1% dead micro-LED pixel, and 20%micro-LED intensity nonuniformity).

AR/VR systems may use a rolling-update display that update the displayedcontent row by row. This is fine when the user's eye is stationary.However, when the user's eye position changes while the frame is beingupdated row by row, the rolling update of the display and the eye motionof the user may cause the human eye to fail to properly perceive thedisplayed image. As a result, the displayed image may appear to bedistorted (e.g., by shearing distortion, vertical compression orexpansion), which may negatively affect the user experience.

In particular embodiments, then AR/VR system may use a correction meshto pre-warp images to counteract the predicted image distortion causedby rolling update and eye motion, before outputting the images todisplay. The system may use an eye tracking system to track the eyevelocity of the user and predict the future eye velocity. After thepredicted eye velocities are determined, the system may generate acorrection mesh, which once applied to the image, can counteract thepredicted image distortions. The correction mesh may include acorrection displacement for each pixel row of the image and eachcorrection displacement, once applied, may counteract the predicteddistortion for that pixel row. The system may first determine a FOVscanning velocity based on the field of view corresponding to the imagearea and the rolling update window period for updating a frame. The FOVscanning velocity may correspond to the rolling update velocity alongthe vertical direction as represented in FOV degrees per second. Then,the system may determine the retina projection velocity based on the FOVscanning velocity and the predicted eye velocity. The retina projectionvelocity may indicate relative velocity of the displayed object beingprojected to the user's retina.

After the retina projection velocity is determined, the system maydetermine the predicted displacement for each pixel row of the imagebased on retina projection velocity and the accumulative time fromupdating the first row until updating that pixel row. The correctiondisplacement in the correction mesh for a particular row may be theinverse of the predicted pixel row displacement. The correctiondisplacement once applied to the corresponding pixel row, may counteractthe predicted pixel row displacement and eliminate or reduce thepredicted distortion (e.g., shearing distortion or vertical expansion orcompression). However, applying correction displacements to image pixelrows may result in an overall displacement or shift for the correctedimage. As a result, the virtual object represented by the correctedimage may have a position displacement with respect to its intendedposition in the FOV of the user. To solve this problem, the system maydetermine an overall displacement based on the correction displacementsof all pixel rules of the image and adjust the displacement for i-th rowby the overall displacement. After the adjusted correction displacementis determined for each pixel row in the correction mesh, the system mayapply the correction mesh to pre-warp the rendered image to counteractthe predicted pixel displacement during the time period when the frameis updated. As a result, the pre-warped image, once displayed, mayeliminate or reduce the rolling update distortion (e.g., shearingdistortions, vertical expansion or compression) and overall positionshift as perceived by the user.

By tracking eye movement using an eye tracking system, particularembodiments of the system may generate an accurate correction mesh thatcan counteract the predicted image distortions caused by the rollingupdate of the display and the eye movement of the user. By generatingand applying the correction mesh to images, particular embodiments ofthe system may improve the user experience by eliminating the imagedistortion caused by the rolling update and the user's eye movement. Bycounteracting the predicted image distortions, particular embodiments ofthe system may improve the quality of the displayed contents to the userby displaying more accurate and precise object shapes. By correcting theoverall displacement, particular embodiments of the system may improvethe quality of the displayed contents by displaying objects at moreaccurate and precise positions with respect to the FOV of the user.

The human visual system may have fine-tuned predictive mechanisms andmay assume a natural, continuous (or nearly continuous) input of photonsfrom surfaces and illumination sources in the real world. Despiteconstant retinal flow due to eye and head movements, the human visualsystem can provide a stable percept of space and motion. In contrast,head mounted displays (HMDs) may be temporally limited which may breakwith the expectations of continuity and create severe perceptualartifacts, for example, perceived flicker. If the global refresh rate ofa display is below the perceptual fusion frequency, and the frameillumination time (persistence) is limited, humans may see flashedstatic images, often interspersed with periods of darkness, instead ofperceiving continuously presented content.

Due to user's eye/head movement, a head mounted display (HMD) may havevisible motion blur in displayed images that may negatively affect theuser experience. Traditional solutions to this problem is to decreaseper-frame the display on-time (known as “persistence” in absolute timeor “duty cycle” in a frame time-relative percentage) to minimize thelight displacement on the retina during user the eye/head motion.However, in LED-based displays, to maintain a constant brightness at lowpersistence levels, the display may consume more power and have reducedlifetime. Furthermore, when the persistence is lower than a threshold,the user may experience other artifacts such as flicker, eye-basedstrobing (also known as “phantom array”) artifacts or decreased eyemovement accuracy.

To solve this problem, particular embodiment of the system may determinean optimized display persistence for each image frame based on imagecontent characteristics (e.g., spatial frequency spectrum, displaycontrast) and eye/head motion velocity of the user, to reduce the motionblur and at the same time avoid flicker. The likelihood of an image tohave visible motion blur may depend on the spatial frequency spectrum ofthe image content. Human visual system may be more sensitive to imagecontent with higher spatial frequencies (e.g., sharp edges). Thus, animage with high spectral spatial frequency content may be more likely tohave visible motion blur. On the other hand, image content with lowerspatial frequencies (e.g., smooth gradient changes) are less likely tohave visible motion blur. The system may compute the spatial frequencyspectrum of an image to be displayed and use a prediction model topredict the likelihood for that image to have visible motion blur andcalculate the corresponding persistence values to void such visiblemotion blur.

In particular embodiments, the system may generate a prediction modelbased on spatial frequency spectrum of image content. The system maycompare the spatial frequency spectrums of the “real world” images andthe “simulated images” output by the display to determine the differenceof the power spectrums that are within the visible range to human visualsystem. The visible range of human visual system may be described by theWindow of Visibility (defined on temporal and spatial frequency domains)which can be used as a spatiotemporal filter during the modelingprocess. The prediction model may include a mapping model as representedby a lookup table that can be stored in memory and accessed at run time.The mapping model may be represented by a look-up table which maps thespatial frequency to corresponding persistence value. At run time, thesystem may access the image to be displayed, compute the spatialfrequency spectrum of that image, and compare the spatial frequencycharacteristics to the stored data in the lookup table to determine theappropriate display persistence for that image. Alternatively, insteadof using a lookup table, the system may use a transfer function tocalculate, at the run time, the corresponding persistence value based onthe predicted the likelihood for that image to have visible motion blur.The transfer function may be pre-determined during the modeling processand may be used to map the spatial frequency spectrum of images tocorresponding persistence values. The system may also use the temporalfrequency characteristics in temporal frequency domain to determine thedisplay persistence.

In particular embodiments, the HMD may use a head tracking system totrack the user's eye/head motion (e.g., motion velocity, motiondirection). The user's eye/head motion velocity is another factor thataffects the likelihood of an image to have visible motion blur. Ingeneral, the faster the user moves his eye/head, the more likely theimage will have visible motion blur. Similar to the spatial frequencyspectrum, the system may generate a prediction model for predictingmotion blur based on the eye/head motion velocity and calculatepersistence values. This prediction model may be generated based onmeasured data that correlates the eye/head motion velocity to theobserved motion blur. The prediction model may include a lookup table ora transfer function. At the runtime, the system may access the look uptable or use the transfer function to determine the display persistencevalue based on the eye/head motion velocity.

In particular embodiments, the images displayed by HMD may havedifferent contrast levels with respect to the background (e.g., an ARobject with respect to real world background) which is another factorthat affects the likelihood for a displayed image to have visible motionblur. The system may use the background-foreground contrast (known asadditive contrast) to predict the likelihood for an image to havevisible motion blur. Similar to the factors of spatial frequencyspectrum and user eye/head motion velocity, the system may generate aprediction model for the additive contrast which may include a lookuptable or a transfer function. The system may use the prediction model todetermine the likelihood for each image frame to have visible motionblur and calculate the corresponding display persistence value for thatimage.

In particular embodiments, the system may use respective predictionmodels for spatial frequency spectrum, user eye/head motion velocity, oradditive contrast to determine the respective display persistence valuesbased on respective factors. Then, the system may determine an optimizeddisplay persistence value based on two or more persistence valuesdetermined by the respective factors. For example, the system may choosethe lower persistence value to make sure the motion blur will bereduced. As another example, the system may use a weighted average asthe optimized persistence value to balance all factors considered. Inanother embodiments, the system may use a single prediction model whichcollectively takes in all factors of spatial frequency spectrum, usereye/head motion velocity, and additive contrast to determine theoptimized display persistence value.

In particular embodiments, after the optimized persistence value isdetermined, the system may configure the display to display thecorresponding image using that optimized persistence value or/and framerate. Particular embodiments of the system may modulate persistence to alow level only when the image content is predicted to likely to havevisible motion blur. As a result, particular embodiments of the systemmay reduce the motion blur artifact and at the same time have a balanceddisplay effect to avoid other artifacts such as flickers. Furthermore,by using low persistence level only for the image that is predicted tohave visible motion blur, particular embodiments of the system mayreduce power consumption and improve display panel lifetime.

FIG. 4 illustrates an example frame updating scheme 400 using periodicaldisplay-on time. As an example and not by way of limitation, the displaymay display a series of images in a discontinuous manner at particularframe rates (e.g., 90 Hz). When the frame rate being used is higher thana threshold (e.g., 80 Hz) corresponding to human visual system fusionfrequency, the human visual system may perceive it as a continuous videorather than discrete and or flickering images. The display may display asingle image during each frame cycle time period 401, which correspondsto the current frame rate. During each frame cycle time period, theimage being display may be displayed only during the display-on timeperiod 402, which may be a portion (e.g., from 0% to 100%) of the framecycle time period 401.

In particular embodiments, the “display-on time” for displaying an imagemay be referred to as “display persistence time,” “display persistence”or “persistence” in absolute value 403 (e.g., 1 ms) or may be referredto as “duty cycle” in percentage 404 with respect to the frame cycletime period 401. In general, the longer the display persistence time,the more likely the displayed image may have visible motion blur. Inparticular embodiments, the system may predict, for each frame of image,the likelihood for that image to have visible motion blur. When an imageto be displayed is predicted to likely to have visible motion blur, thesystem may reduce the persistence time for that image to avoid themotion blur. When an image to be displayed is predicted to unlikely tohave visible motion blur, the system may use the same or longer displaypersistence time for that image. In other words, the system may modulatepersistence to a low level only when the image content is predicted tolikely to have visible motion blur. As a result, the system may reducethe motion blur artifact and at the same time have a balanced displayeffect to avoid other artifacts such as flickers.

In particular embodiments, motion blur may be a highly salientspatiotemporal artifact that drastically reduces the user experience.Motion blur may occur when an image is displayed with a finite refreshrate of the display and the user's eyes move across the image. Due tomovement of the user's eyes, the stationary image may be moved acrossthe retina and the spread out information may be integrated into ablurry percept. Motion blur may be highly salient to users and alsocause users to have reduced visual acuity to the blurred image. Headmounted displays (HMDs) may be especially prone to the percept of motionblur, since the image needs to be continuously updated with respect tohead movements to remain apparently world-fixed. Head movements may beaccompanied by smooth rotations of the eye into the opposite directions(the vestibulo-ocular reflex or VOR) to stabilize the image on theretina. However, since HMDs may run at limited refresh rates (e.g. 72Hz), the display images may have substantial motion blur due to thedisplacement of images across the retina, especially during VOR. Toreduce motion blur, HMDs may run at a low duty cycle (the proportion ofthe frame the display is illuminated) to reduce the displacement that isintroduced on the retina image.

FIG. 5 illustrates an example Window of Visibility 501 for human visualsystem. In particular embodiments, the Window of Visibility 501 mayprovide the spatial frequency range and the temporal frequency rangethat are visible to human visual system (e.g., human visual system has asensitivity level above a threshold level in these spatial frequencyrange and temporal frequency range). For example, the Window ofVisibility 501 may indicate that the spatial frequencies under u₀cycles/deg in the spatial frequency domain is visible to human visualsystem and the temporal frequencies under u₀ Hz in the temporalfrequency domain is visible to human visual system. The energy beyondthe ranged as defined by the Window of Visibility 501 may not be visibleto human visual system.

FIG. 6A illustrates an example process 600A for using a prediction model612 to predict the likelihood for an image to have artifacts and todetermine the corresponding persistence value. In particularembodiments, the system may use the information provided by the Windowof Visibility to assess or predict the likelihood of occurrence ofparticular type of artifacts. It is notable that, this may be applicableto both grayscale images and chromatic images. In particularembodiments, the system may use known spatiotemporal sensitivities ofthe human visual system (e.g., as described by the Window of Visibility)to assess or/and predict the likelihood of occurrence of artifacts(e.g., flickers, motion blur) across different display characteristicsor/and cross different VR and AR contents.

As an example and not by way of limitation, the system may simulate areal world scene 601 (e.g., using a real world image or the real worldsense itself) with continuous input and compare its spatiotemporal powerspectrum with the spatiotemporal power spectrum of a “simulated image”as displayed by a display 602. The system may use a learning framework609 to determine the correlations between certain image characteristics(in the spatiotemporal power spectrum) and the artifacts in the image.The framework 609 may include spatiotemporal power spectrum modules 603and 604 and a comparator 607 to determine the difference in thespatiotemporal power spectrum. The system may use the spatiotemporalpower spectrum analyzing module 603 to compute the spatiotemporal powerspectrum for the real world scene.

The system may use the spatiotemporal power spectrum analyzing module604 to compute the spatiotemporal power spectrum for the displayedimage. Then, the system may use the comparator 607 to compare thespatiotemporal power spectrums of the real word scene 601 and thedisplayed image 602 to determine their difference in the spatiotemporalpower spectrums. In particular embodiments, the system may focus on thecomparison of the spatiotemporal power spectrums that fall within thevisible range to human visual system as described by the Window ofVisibility. In particular embodiments, the system may focus on thecomparison on the image content that are within a foveated displayregion enclosing the gazing point of the user.

After determining the difference 605 of the spatiotemporal powerspectrums of the real world scene and the simulated image, the systemmay identify the characteristics of the image content in thespatiotemporal power spectrum that are linked to the artifacts 606. Forlearning and training purpose, to determine whether a simulated imagehas artifacts and what types of artifacts (e.g., motion blur, flicker)for the ground truth data the image may be examined by human users orcomputer algorithms (e.g., the effect as a single image being displayedand the effect as displayed within a series of images).

Using the framework 609, the system may learn the knowledge of the imagecontent characteristics in the spatiotemporal power spectrum of imagesthat are linked to the artifacts. Then, the system may use thisknowledge to generate the prediction model 612. For example, the systemmay determine one or more rules based on the image characteristics thatare indicative to the likelihood for that image to have artifacts anduse these rules to create the mapping model 614 that can be used by theprediction model 612 to predict the likelihood of an image to haveartifacts and to calculate the persistence value. As another example,the mapping model 614 may be or include one or more machine-learningmodels and the system may use the knowledge with respect to theindicative characteristics to identify or generate a number of trainingsamples that can be used to train the machine learning model to predictthe likelihood of the artifacts and to determine the persistence values.

At run time, the system may use the prediction model 612 to analyze eachframe of image to be displayed to predict the likelihood for that imageto have artifacts. For example, the system may use the spatiotemporalpower spectrum analyzing module 613 to compute the spatiotemporal powerspectrum of the image content of the image to be displayed 611. Then,the system may feed the spatiotemporal power spectrum of the imagecontent to the mapping model 614, which may use, for example, but notlimited to, a lookup table, a transfer function, one or more rule-basedlogics, or one or more machine-learning models, to predict thelikelihood of that image to have artifacts, the types of artifacts(e.g., motion blur, strobing) that are predicted to have, and todetermine the corresponding persistence values that can avoid thepredicted artifacts.

In particular embodiments, to assure that the difference in thespatiotemporal power spectrums is related to artifacts, the system maygenerate a simulation of the real world scene (e.g., using a displayedimage on a display) which has essentially the same brightness(essentially the same overall photon count) to the real world scene. Inan empirical sense, the system may implement the possibility of a directcomparison, for example in a two-alternative forced choice (2AFC) taskbetween a real world stimulus and the display to avoid confounds byadditional differences in brightness. Matching the brightness may beachieved by matching the lighting in the room to the properties of thedisplay. Due to the temporal limitations, the simulated display maypresent content for a fraction of time of the continuously illuminatedreal world. The instantaneous brightness of the simulated display mayneed to be higher. The brightness levels of the real world scene and thesimulated display may be expressed using the following equations:

$\begin{matrix}{{instbright_{real}} = \frac{Brightness}{Time}} & (1)\end{matrix}$ $\begin{matrix}{{{inst}bright_{disp}} = \frac{Brightness}{{Time}({Illuminated})}} & (2)\end{matrix}$

where, instbright_(real) is the instantaneous brightness level of thereal world scene; instbright_(disp) is the instantaneous brightnesslevel of the simulated display; Time is the duration of a givensimulation interval.

In particular embodiments, the fraction of time that the display isilluminated may depend on different factors related to the displayarchitecture. For a globally-illuminated display, the fraction of timethat the display is illuminated may depend on the refresh rate and dutycycle of the display and there may be more complex interactions forother display architectures. With the instantaneous brightness level,the system may calculate the photon count from the image pixel valuesfor a given display contrast. For example, based on the contrastdefinition in Equation (3), the system may convert the luminance valuesof the pixels of an image into a proxy of a photon count based onEquation (4) as follows:

contrast=1+bright_(Foreground)/bright_(Background)   (3)

photoncount=phot_(low)+(phot_(high)−phot_(low))*Lum_(Pixel)   (4)

where, photoncount is the calculated photon count from the image pixelvalue for a given display contrast; phot_(low) equals toinstbright/√{square root over (Contrast−1)}and phot_(high) equals toinstbright*√{square root over (Contrast−1)}; bright_(Foreground) andbright_(Background) are the brightness levels of the foreground imageand the background for displaying this image.

In particular embodiments, the system may simulate an additive ARbackground. The system may determine the instantaneous brightness of agiven contrast to the display using the following equation:

$\begin{matrix}{{instbright_{Additive}} = \frac{{Brightness}/\left( {{AddContrast} - 1} \right)}{Time}} & (5)\end{matrix}$

where, instbright_(Additive) is the instantaneous rightness of theadditive background; AddContrast is the contrast level between the imagecontent and the addictive background. Based on these values, the systemmay simulate the real world and display with matched brightness levelswith (for an AR-like display) or without (for an VR-like display)additive background.

In particular embodiments, due to the organization of human retinalphotoreceptors and visual system, human visual system may only haveaccess to the highest amount of spatial detail close to the center ofvisual field and the sensitivity may drastically decrease towards theperiphery. To mimic this effect, the system may build a spatial filterthat mimics this drop of sensitivity in the periphery. The sensitivityto a given spatial frequency at a certain eccentricity may be estimatesusing the following equation:

sensitivity=a+(b*(1+c*eccentricity))*frequency   (6)

where, sensitivity is the calculated visual sensitivity of human visualsystem; the estimated parameters of a, b, and c are as following:a=1.82, b=−0.079 and c=0.28; eccentricity is the distance in visualdegree; frequency is the spatial frequency. In particular embodiments,the system may decompose the image into different frequency bands anduse the sensitivity to filter these frequencies with the giveneccentricity. The instances where sensitivity of spatial frequencies ata given eccentricity drops below zero may be removed, since these maynot be visible to the observer. The sensitivities of all visiblefrequencies may be normalized and used to weight the different frequencybands for a given point in the image.

In particular embodiments, after the simulation of the display iscompleted, the system may generate a 3D matrix (e.g., with the threedimensions of horizontal pixels, vertical pixels, and simulation steps)and compute a 3D Fourier transform. To match the output with the Windowof Visibility, which defines the spatiotemporal limitations of humanperception, the system may radially average two spatial dimensions foreach temporal frequency to obtain a 2D matrix reflecting thespatiotemporal power spectrum.

After obtaining the spatiotemporal power spectrums for the real worldscene and display simulation, the system may determine the differencebetween the spatiotemporal power spectrums of the real world scene anddisplay simulation. The differences in the visible parts of the powerspectrum may provide information about the likelihood, the amount, andthe type of the possible perceptual artifact. In order to generalizethese measures across different images, the system may normalize thedifference in the spatiotemporal power spectrum with the overall photoncount as mainly relative contrasts and differences that are relevant forthe visual system.

In particular embodiments, the system may use the above describedmodeling framework to learn the knowledge of the image contentcharacteristics in the spatiotemporal power spectrum. Then, the systemmay determine links between these characteristics (e.g., high spatialfrequency components) and the predicted artifacts (e.g., motion blur).After that, the system may use this knowledge to generate the predictionmodel that can be used to predict the likelihood for images to haveartifacts and calculate the persistence values. In particularembodiments, the prediction model may include a mapping model asrepresented by a look-up table or a transfer function to map the inputfactors (e.g., image content spatial frequency, user head motionvelocity, additive contrast) to corresponding persistence value. Themapping model may be generated based on one or more rules related to theimage characteristics and their links to the predicted artifacts. Inparticular embodiments, the prediction model may include one or moremachine-learning models that are trained based on the knowledge relatedto the indicative characteristics and the likelihood of having artifactsto predict the artifacts and determine the persistence values.

FIG. 6B illustrates an example mapping model 614 which maps the inputfactors to corresponding persistence values. In particular embodiments,the mapping model 614 may include a look-up table 621, a transferfunction 622, or a machine-learning model 623 each of which may map theinput factor 624 to a corresponding persistence value 625. It is notablethat the mapping model 614 may not need to have one or morerepresentations selected from the look-up table 621, the transferfunction 622, or the machine-learning model 623 and does not need toinclude all of them. The look-up table 621 may be generated during themodeling process based on the experiential knowledge or/and experimentaldata obtained during the modeling process. The look-up table may bestored in the memory and accessed by the system at run time to determinethe persistence value based on the input factor. In particularembodiments, to determine the display persistence time period, thesystem may access the lookup table that maps the one or morecharacteristics (e.g., spatial frequency of the content of the image) tocorresponding display persistence time periods. The system may comparethe image content characteristics (e.g., the spatial frequency) tostored data in the lookup table to determine the display persistencetime period. The look-up table may allow the system to reduce powerconsumption and provide fast results with minimum amount of computation.

In particular embodiments, the transfer function 622 may generated basedon the experiential knowledge or/and experimental data obtained duringthe modeling process and may use directly by the system to calculate thepersistence time at run time. In particular embodiments, the system mayuse a difference to map the results to psychophysical data set. Bysimulation of the visual content during the psychophysical experiment,the system may generate a difference map for each psychophysical datapoint. Then, the system may use these data points to build a transferfunction 622 that maps the difference onto psychophysical data. Withthis transfer function, the system may predict the percept of thedisplayed image with various simulation parameters and the displaypersistence time period may be determined using a transfer function onrun time. The transfer function 622 may allow the system to have moreflexibility (e.g., easy to be updated) and keep the amount ofcomputation relatively small.

In particular embodiments, the machine-learning model 623 may be trainedbased on the experiential knowledge or/and experimental data obtainedduring the modeling process to predict the artifacts and determine thecorresponding persistence values. In particular embodiments, the inputfactor 624 may be, for example, but are not limited to, an image contenttemporal or spatial frequency, a user head motion velocity, or abackground contrast level. In particular embodiments, the displaypersistence time period for the image may be or include an absolute timeperiod value or a display duty cycle in percentage.

In particular embodiments, after the prediction model has been created,the system may, at run time and before displaying each frame of image,use the prediction model to analyze the current frame of image to bedisplayed to predict the likelihood of that image to have artifacts. Thesystem may compute the spatiotemporal power spectrum for the content ofthe current frame of image. Then, the system may determine one or morecharacteristics associated with the image content based on thespatiotemporal power spectrum. The characteristics of the image contentmay include, for example, but are not limited to, spatial frequencycomponents higher than a threshold spatial frequency, spatial frequencycomponents lower than a threshold spatial frequency, temporal frequencycomponents higher than a temporal frequency threshold, temporalfrequency components lower than a temporal frequency threshold,spatiotemporal power components falling with a particular spatiotemporalfrequency range, etc. Then, the system may use the prediction models(e.g., based on one or more rules) or machine-learning models (trainedbased on historical data and knowledge) to predict (1) the likelihoodfor that image to have artifact, and (2) the type and amount ofartifacts that are predicted to happen and to determine thecorresponding persistence time values.

FIGS. 7A-7B illustrate example images that have content with differentspatial frequencies and allow the display to use different displaypersistence values. In particular embodiments, the amount of motion blurperception may depend on display refresh rates and duty cycles of thedisplay. The critical duty cycle to perceive motion blur may differacross different refresh rates. Thus, from the display perspective,motion blur may be mainly driven by display persistence (i.e., the timethe display is illuminated during each frame cycle). From the imagecontent perspective, the likelihood to have visible motion blur maydepend on the spatial frequency of the image content. In particularembodiments, the image content may play an important role in theprobability of perceiving motion blur. On one hand, image content thathas higher spatial frequencies (e.g., sharp contrast edges, text) may bemore likely to have visible motion blur and, thus may require shorterdisplay persistence values to avoid motion blur. On the other hand,image content that has lower spatial frequencies (e.g., naturalisticimage content such as a picture of desert night with moon) may be lesslikely to have visible motion blur, and thus allow the display to uselonger display persistence values.

In particular embodiments, the system may determine a displaypersistence time period for an image based on the one or morecharacteristics in the spatiotemporal power spectrum of the imagecontent of that image. As an example and not by way of limitation, theimages 700A, 700B, and 700C (corresponding to image Number 1, Number 2and Number 3) may have content with different spatial frequencies. Theimage content of the image 700A may have higher spatial frequencies thanthe image 700B, which may have higher spatial frequencies than the image700C. As a result, the three images of 700A, 700B, and 700C may needdifferent display persistence values as shown in FIG. 7B. For someimages having high spatial frequency content (e.g., 700A), the image mayneed to be displayed with a relatively short persistence time period(e.g., 1 ms) to avoid motion blur. For the images having lower spatialfrequency content (e.g., 700B and 700C), the persistence time periodsthat are need may be much longer (e.g., about 4 ms for image 700B, andabout 11 ms for image 700C). Thus, the system may adaptively increase ordecrease the persistence time periods based on the image contentcharacteristics. After that, the system may configure the display todisplay the images using corresponding display persistence time periods.The display persistence time period may reduce or eliminate the motionblur artifacts that are predicted to happen on these images. Inparticular embodiments, the spatial frequency of the content may behigher than a preceding image and the display persistence time periodfor the image may be shorter than a previous display persistence timeperiod for the preceding image to reduce the motion blur artifacts. Inparticular embodiments, the spatial frequency of the content may belower than a preceding image and the display persistence time period forthe image may be longer than a previously display persistence timeperiod of the preceding image.

In particular embodiments, the system may use the Window of Visibilityof human visual system to filter the image content in the spatiotemporaldomain. The Window of Visibility of human visual system may describe thespatiotemporal frequency range that are visible to human visual systemin the spatiotemporal domain. The system may only analyze the contentpower spectrum portions that fall within the Window of Visibility ofhuman visual system and ignore the content power spectrum portions thatfall outside the Window of Visibility of human visual system (becausethey are not visible to human visual system). The image contentcharacteristics that linked to artifacts may fall within the Window ofVisibility. By excluding the content power spectrum portions that falloutside the Window of Visibility, the system may effectively identifythe image content characteristics that are indicative for predictingartifacts, and at the same time, reduce the amount of computation andpower consumption needed for performing these tasks.

In particular embodiments, the head motion velocity of the user duringVOR may be one of the main factors that influence the probability ofperceived motion blur. The system may use a head motion system (e.g.,inertial measurement unit (IMU) of HMD, camera-based tracking system, orrigid-body-based tracking system or an eye tracking system) to determinethe head motion or eye motion of the user. For faster movements of theuser's head, the chances of perceiving motion blur may increase, and themagnitude of motion blur may increase as well. In particularembodiments, a persistence of around 1 ms may be needed to achieveacceptable blur perception levels for head motion velocities above 50deg/s. For lower head motion velocities, the persistence may beincreased up to about 2 ms. For head motion velocities during everydaytasks, activities having head velocities above 50 deg/s may be less than75%. Thus, for a large proportion of everyday motion of the users, thesystem may keep the time persistence at a relative high level.

In particular embodiments, the system may determine, using a headtracking system or an eye tracking system, a head motion velocity or aneye motion velocity of the user. The system may determine a displaypersistence time period for the image based on the head motion velocityof the user. The system may determine the display persistence timeperiod using a prediction model, which may include a lookup table, atransfer function, or a machine-learning model. The prediction model maybe generated using the modeling framework as described in the earliersections based on the knowledge learned from measurement andexperimental data. For example, the prediction model may include alookup table or a transfer function that maps the user head motionvelocity to corresponding display persistence value. The lookup table orthe transfer function may be generated based on the experiential orexperimental data. As another example, the prediction model may includea machine-learning model that is trained based on the experiential orexperiment data. In particular embodiments, system may determine anoptimized persistence time period considering both image content and theuser head motion velocity. The optimized display persistence time periodmay be a shorter time period selected from the display persistence timeperiod determined based on the image content and the display persistencetime period determined based on the user head motion velocity. Inparticular embodiments, the optimized display persistence time periodmay be a weighted average of the display persistence time periodsdetermined based on the image content and the user head motion velocity,respectively.

In particular embodiments, for VR displays, the system may have fullcontrol of the lighting conditions. However, AR glasses may allow theuser to still see the real world, which provides a dynamically changingbackground. The system may use the prediction model to predict that theadditive contrast of the display with respect to the background, as wellas to some extent the background content affect the perception of motionblur. When the contrast is low, the background may mask some of theoccurring motion blur. In contrast, when the contrast is high or nobackground for a VR display, the system may have an increased chance tohave visible motion blur perception.

In particular embodiments, the system may determine a contrast level ofthe content of the image with respect to a background of the image. Thesystem may determine the contrast level of the image content withrespect to the background by accessing the pixel values of the imagecontent and an image of the background (or by measuring the backgrounddirectly using light sensors). Then, the system may determine a displaypersistence time period for the image based on the contrast level of thecontent of the image with respect to the background of the image. Thepersistence value may be determined using a prediction model generatedusing the modeling framework. In particular embodiments, the predictionmodel may include a lookup table or a transfer function that can mapbackground contrast levels to corresponding persistence values. Thelookup table or the transfer function may be generated using themodeling framework based on the experiential or experimental data. Inparticular embodiments, the prediction model may be or include amachine-learning model which is trained based on experiential orexperimental data to predict the

In particular embodiments, the system may determine an optimized displaypersistence time based on the display persistence time period determinedbased on image content and a second persistence display time perioddetermined based on background contrast. The optimized displaypersistence time may the shorter persistence time period selected fromthe display persistence time period determined based on the content andthe second persistence display time period determined based on thecontrast level of the content with respect to the background of theimage. In particular embodiments, the optimized persistence value may bea weighted average of the two persistence time periods as determinedbased on the image content and the background contrast level. Afterthat, the system may configure the display to display the image usingthe optimized display persistence time period.

Due to the organization of human visual system, human visual system maybe most sensitive to high spatial frequencies at the areas close to thecenter of the visual field and sensitivity may decreases towards theperiphery. Therefore, even when an image contains high spatialfrequencies, if the user looks at a point of the image with lessdetails, the probability of motion blur detection may decrease becausethe content with high spatial frequencies may be within the areas thatthe user's eyes are not sensitive to.

In particular embodiments, the system may use an eye tracking system todetermine the gazing point of the user and determine a foveated regionfor the image content. For the image content associated within thefoveated region encompassing a gazing point of the user, the system mayanalyze the spatiotemporal power spectrum to of the image content topredict the possible artifacts and determine the persistence values. Forthe image content falling outside the foveated region, the system mayexclude that from the analysis for determining the image contentcharacteristics to predict the artifacts and determine the persistencevalues. In particular embodiments, the system may use distances to thegazing point as weighing factor to weight the image content forpredicting the artifacts and determining the persistence values. Theimage content that is closer to the gazing may be weighted more than thecontent that more peripheral to the gazing point.

FIG. 8 illustrates an example process 800 for determining an optimizedpersistence value based on multiple factors. In particular embodiments,the system may determine an optimized display persistence value 841based on two or more factors (e.g., 831A, 831B, 831C) including, forexample, but not limited to, the spatial frequency in the spatialfrequency domain, a head motion velocity or eye motion velocity, or acontrast level between the image content and a background. The systemmay generate a mapping model for each factor separately and use thatprediction model for that particular factor. In particular embodiments,the system may determine the display persistence values separately basedon respective factors (e.g., 831A, 831B, 831C) using differentprediction models (e.g., 830A, 830B, 830C) and determine an optimizeddisplay persistence value 841 based on the separately determined displaypersistence values (e.g., 831A, 831B, 831C) using an optimization module840. The optimization module 840 may determine the optimized persistencevalue 841 using different methods, for example, by selecting the shorteror shortest ones or using a weighted average as the optimizedpersistence value 841.

In particular embodiments, the system may determine the optimizedpersistence time based on: a first display persistence time perioddetermined based on the one or more image content characteristics (e.g.,the spatial frequency in the spatial frequency domain), a second displaypersistence time period determined based on a head motion velocity,or/and (3) a third display persistence time period determined based on acontrast level between the content of the image and a background and thegazing position of the user on the current image. In particularembodiments, the optimized persistence display time period may be ashorter display persistence time period selected from two displaypersistence time periods determined based on respective factors (e.g.,any two factors of content spatial frequency, user head motion velocity,or background contrast, etc.). In particular embodiments, the optimizedpersistence display time period may be the shortest display persistencetime period selected from a number of the display persistence timeperiods determined based on respective factors (e.g., content spatialfrequency, user head motion velocity, background contrast, etc.). Forexample, the optimized persistence time period may be the shortestdisplay persistence time period selected from the first, second, andthird display persistence times determined based on respective factorsof image content (spatial frequency), user head motion velocity (or eyemotion velocity), and the background contrast level. In particularembodiments, the optimized display persistence time period may be aweighted average of the first, second, and third display persistencetime periods. In particular embodiments, the system may generate aprediction model that can handle multiple factors at the same time andthe system may use that prediction model to determine an optimizeddisplay persistence time period based on all factors that are consideredat the same time.

In particular embodiments, the display persistence time period may bedetermined based on a time period threshold to avoid flicker artifactsand the display persistence time period may be longer than the timeperiod threshold below which the displayed image would likely haveflicker artifacts. Changing the display persistence may have othereffects on the displayed images. On one hand, higher persistence of thedisplay may allow the system to use lower currents to drive the displayto achieve the same brightness, which can increase the lifetime of thepanel. On the other hand, lower persistence may lead to other perceptualartifacts (e.g., flickers) and even impact the oculomotor behavior. Inparticular embodiments, the system may consider a number of factors inthe circumstances that can potentially influence the image artifacts,the displayed image quality, and the display panel lifetime to determinethe optimal display persistence time. The system may predict thelikelihood for images to have different artifacts and determine anadaptive display persistence dynamically to reduce the artifacts,considering a wide range of factors (e.g., other artifacts, powerconsumption, lifetime of the display panel, etc.). After the optimizeddisplay persistence time is determined, the system may configure thedisplay to display the image using the optimized display persistencetime period.

In particular embodiments, when an image has content with high spatialfrequencies, the image may be more likely to have visible motion blurthan an image having low spatial frequency content and the system mayadopt a shorter persistence time. However, if the frame rate is fixed, ashorter persistence may lead to a dimmer result in the displayed image.The system may increase the frame rate to keep the brightness level andalso to avoid flicker artifacts. In particular embodiments, the systemmay determine a frame rate for displaying the image based on the one ormore characteristics of the image content. The characteristics of theimage content may be in the spatial frequency domain (e.g., the spatialfrequency) or/and in the temporal frequency domain (e.g., the temporalfrequency). The characteristics may include, for example, but are notlimited to, spatial frequency components higher than a threshold spatialfrequency, spatial frequency components lower than a threshold spatialfrequency, temporal frequency components higher than a temporalfrequency threshold, temporal frequency components lower than a temporalfrequency threshold, spatiotemporal power components falling with aparticular spatiotemporal frequency range, etc. Then, the system mayconfigure the display to display the image using the frame rate.

In particular embodiments, the system may determine the frame rate fordisplaying an image based on the user head motion velocity or eye motionvelocity. In general, when the user's head or eye moves with a highervelocity, the user may be more likely to perceive motion blur. Thesystem may adopt the shorter persistence and, at the same time, a higherframe rate to void the motion blur and possible flickers. On the otherhand, when the user's head or eye moves with a lower velocity, the usermay be less likely to perceive motion blur. The system may adopt thelonger persistence and a lower frame rate.

In particular embodiments, the system may determine the frame rate fordisplaying an image based on the contrast level between the imagecontent and the background for the image to be displayed in. In general,when the image content and the background have a higher contrast level,the user may be more likely to perceive motion blur. The system mayadopt the shorter persistence and, at the same time, a higher frame rateto void the motion blur and possible flickers. On the other hand, whenthe image content and the background have a lower contrast level, theuser may be less likely to perceive motion blur. The system may adoptthe longer persistence and a lower frame rate.

In particular embodiments, the system may determine an optimized framerate for displaying the images based on multiple factors. In particularembodiments, the system may determine a frame rate for displaying imagesbased on image content (spatial frequency), user head velocity, andimage content background contrast, respectively, and determine anoptimized frame rate based on the separately determined frame rates. Forexample, the optimized frame rate may be a weighted average of theseparately determined frame rates. As another example, the optimizedframer rate may be the highest or lowest frame rate selected from theframe rates determined based on image content (spatial frequency), userhead velocity, and image content background contrast, respectively.After the optimized frame rate is determined, the system may configurethe display to display the corresponding images using the optimizedframe rate.

FIG. 9 illustrates an example method 900 for adaptively determiningdisplay persistence time based on image content. The method may begin atstep 910, where a computing system may access an image to be displayedby a display. At step 920, the system may determine one or morecharacteristics associated with a content of the image. The one or morecharacteristics may include a spatial frequency of the content in aspatial frequency domain. For example, one characteristic may be thatthe image content has spatial frequency components that are higher thana spatial frequency threshold and the image may have a high likelihood(e.g., higher than a threshold) to have visible motion blur artifacts inthis situation. At step 930, the system may determine a displaypersistence time period for the display to display the image based onthe one or more characteristics associated with the content of theimage. The display persistence time period may reduce or eliminate themotion blur artifacts that are predicted to happen on this image. Atstep 940, the system may configure the display to display the imageusing the display persistence time period.

In particular embodiments, the spatial frequency of the content may behigher than a pre-determined spatial frequency threshold in a spatialfrequency domain. In particular embodiments, the spatial frequency ofthe content may be within a pre-determined spatial frequency range(e.g., a visible range to human visual system) in the spatial frequencydomain. The predetermined spatial frequency range may be associated witha Window of Visibility describing a visible range of human visual systemin the spatial frequency domain. In particular embodiments, the spatialfrequency of the content may be higher than a preceding image and thedisplay persistence time period for the image may be shorter than aprevious display persistence time period for the preceding image toreduce the motion blur artifacts.

In particular embodiments, the spatial frequency of the content may belower than a preceding image and the display persistence time period forthe image may be longer than a previously display persistence timeperiod of the preceding image. In particular embodiments, the one ormore characteristics including the spatial frequency of the content ofthe image may be determine based on a spatiotemporal power spectrum ofthe content of the image. In particular embodiments, the content of theimage may be associated within a foveated region encompassing a gazingpoint of the user, and a second content of the image falling outside thefoveated region may be excluded from determining the one or morecharacteristics associated with the content of the image. In particularembodiments, the image content may be weighted by the correspondingdistances to the gazing point within the foveated region to determinethe likelihood to have visible motion blur and the corresponding displaypersistence time period to avoid the motion blur. In particularembodiments, the display persistence time period for the image may be orinclude an absolute time period value or a display duty cycle inpercentage.

In particular embodiments, the system may determine, using a headtracking system, a head motion velocity of a user. The system maydetermine a second display persistence time period for the image basedon the head motion velocity of the user. The system may determine anoptimized display persistence time period based on the displaypersistence time period and the second display persistence time period.The system may configure the display to display the image using theoptimized display persistence time period. In particular embodiments,the optimized display persistence time period may be a shorter timeperiod selected from the display persistence time period and the seconddisplay persistence time period. In particular embodiments, theoptimized display persistence time period may be a weighted average ofthe display persistence time period and the second display persistencetime period.

In particular embodiments, the system may determine a contrast level ofthe content of the image with respect to a background of the image. Thesystem may determine a second display persistence time period for theimage based on the contrast level of the content of the image withrespect to the background of the image. The system may determine anoptimized display persistence time based on the display persistence timeperiod and the second persistence display time period. The optimizeddisplay persistence time may the shorter persistence time periodselected from the display persistence time period determined based onthe content and the second persistence display time period determinedbased on the contrast level of the content with respect to thebackground of the image. The system may configure the display to displaythe image using the optimized display persistence time period.

In particular embodiments, to determine the display persistence timeperiod, the system may access a lookup table that maps the one or morecharacteristics including the spatial frequency of the content of theimage to corresponding display persistence time periods. The system maycompare the one or more characteristics including the spatial frequencyof the content of the image to stored data in the lookup table todetermine the display persistence time period. In particularembodiments, the display persistence time period may be determined usinga transfer function on run time. In particular embodiments, the systemmay determine one or more second characteristics associated with thecontent of the image. The one or more second characteristics may includea temporal frequency in a temporal frequency domain. The system maydetermine a frame rate for the image based on the one or more secondcharacteristics associated with the content of the image. Then, thesystem may configure the display to display the image using the framerate. In particular embodiments, the frame rate may be determined basedon the one or more characteristics including the spatial frequency inthe spatial frequency domain and the one or more second characteristicsincluding the temporal frequency in the temporal frequency domain.

In particular embodiments, the system may determine an optimized displaypersistence time period based on (1) a first display persistence timeperiod determined based on the one or more characteristics comprisingthe spatial frequency in the spatial frequency domain, (2) a seconddisplay persistence time period determined based on a head motionvelocity, and (3) a third display persistence time period determinedbased on a contrast level between the content of the image and abackground. In particular embodiments, the optimized persistence displaytime period may be the shortest display persistence time period selectedfrom the first, second, and third display persistence times determinedbased on respective factors. In particular embodiments, the optimizeddisplay persistence time period may be a weighted average of the first,second, and third display persistence time periods. Then, the system mayconfigure the display to display the image using the optimized displaypersistence time period. In particular embodiments, the displaypersistence time period may be determined based on a time periodthreshold to avoid flicker artifacts and the display persistence timeperiod may be longer than the time period threshold.

Particular embodiments may repeat one or more steps of the method ofFIG. 9, where appropriate. Although this disclosure describes andillustrates particular steps of the method of FIG. 9 as occurring in aparticular order, this disclosure contemplates any suitable steps of themethod of FIG. 9 occurring in any suitable order. Moreover, althoughthis disclosure describes and illustrates an example method foradaptively determining display persistence time based on image contentincluding the particular steps of the method of FIG. 9, this disclosurecontemplates any suitable method for adaptively determining displaypersistence time based on image content including any suitable steps,which may include all, some, or none of the steps of the method of FIG.9, where appropriate. Furthermore, although this disclosure describesand illustrates particular components, devices, or systems carrying outparticular steps of the method of FIG. 9, this disclosure contemplatesany suitable combination of any suitable components, devices, or systemscarrying out any suitable steps of the method of FIG. 9.

FIG. 10 illustrates an example computer system 1000. In particularembodiments, one or more computer systems 1000 perform one or more stepsof one or more methods described or illustrated herein. In particularembodiments, one or more computer systems 1000 provide functionalitydescribed or illustrated herein. In particular embodiments, softwarerunning on one or more computer systems 1000 performs one or more stepsof one or more methods described or illustrated herein or providesfunctionality described or illustrated herein. Particular embodimentsinclude one or more portions of one or more computer systems 1000.Herein, reference to a computer system may encompass a computing device,and vice versa, where appropriate. Moreover, reference to a computersystem may encompass one or more computer systems, where appropriate.

This disclosure contemplates any suitable number of computer systems1000. This disclosure contemplates computer system 1000 taking anysuitable physical form. As example and not by way of limitation,computer system 1000 may be an embedded computer system, asystem-on-chip (SOC), a single-board computer system (SBC) (such as, forexample, a computer-on-module (COM) or system-on-module (SOM)), adesktop computer system, a laptop or notebook computer system, aninteractive kiosk, a mainframe, a mesh of computer systems, a mobiletelephone, a personal digital assistant (PDA), a server, a tabletcomputer system, an augmented/virtual reality device, or a combinationof two or more of these. Where appropriate, computer system 1000 mayinclude one or more computer systems 1000; be unitary or distributed;span multiple locations; span multiple machines; span multiple datacenters; or reside in a cloud, which may include one or more cloudcomponents in one or more networks. Where appropriate, one or morecomputer systems 1000 may perform without substantial spatial ortemporal limitation one or more steps of one or more methods describedor illustrated herein. As an example and not by way of limitation, oneor more computer systems 1000 may perform in real time or in batch modeone or more steps of one or more methods described or illustratedherein. One or more computer systems 1000 may perform at different timesor at different locations one or more steps of one or more methodsdescribed or illustrated herein, where appropriate.

In particular embodiments, computer system 1000 includes a processor1002, memory 1004, storage 1006, an input/output (I/O) interface 1008, acommunication interface 1010, and a bus 1012. Although this disclosuredescribes and illustrates a particular computer system having aparticular number of particular components in a particular arrangement,this disclosure contemplates any suitable computer system having anysuitable number of any suitable components in any suitable arrangement.

In particular embodiments, processor 1002 includes hardware forexecuting instructions, such as those making up a computer program. Asan example and not by way of limitation, to execute instructions,processor 1002 may retrieve (or fetch) the instructions from an internalregister, an internal cache, memory 1004, or storage 1006; decode andexecute them; and then write one or more results to an internalregister, an internal cache, memory 1004, or storage 1006. In particularembodiments, processor 1002 may include one or more internal caches fordata, instructions, or addresses. This disclosure contemplates processor1002 including any suitable number of any suitable internal caches,where appropriate. As an example and not by way of limitation, processor1002 may include one or more instruction caches, one or more datacaches, and one or more translation lookaside buffers (TLBs).Instructions in the instruction caches may be copies of instructions inmemory 1004 or storage 1006, and the instruction caches may speed upretrieval of those instructions by processor 1002. Data in the datacaches may be copies of data in memory 1004 or storage 1006 forinstructions executing at processor 1002 to operate on; the results ofprevious instructions executed at processor 1002 for access bysubsequent instructions executing at processor 1002 or for writing tomemory 1004 or storage 1006; or other suitable data. The data caches mayspeed up read or write operations by processor 1002. The TLBs may speedup virtual-address translation for processor 1002. In particularembodiments, processor 1002 may include one or more internal registersfor data, instructions, or addresses. This disclosure contemplatesprocessor 1002 including any suitable number of any suitable internalregisters, where appropriate. Where appropriate, processor 1002 mayinclude one or more arithmetic logic units (ALUs); be a multi-coreprocessor; or include one or more processors 1002. Although thisdisclosure describes and illustrates a particular processor, thisdisclosure contemplates any suitable processor.

In particular embodiments, memory 1004 includes main memory for storinginstructions for processor 1002 to execute or data for processor 1002 tooperate on. As an example and not by way of limitation, computer system1000 may load instructions from storage 1006 or another source (such as,for example, another computer system 1000) to memory 1004. Processor1002 may then load the instructions from memory 1004 to an internalregister or internal cache. To execute the instructions, processor 1002may retrieve the instructions from the internal register or internalcache and decode them. During or after execution of the instructions,processor 1002 may write one or more results (which may be intermediateor final results) to the internal register or internal cache. Processor1002 may then write one or more of those results to memory 1004. Inparticular embodiments, processor 1002 executes only instructions in oneor more internal registers or internal caches or in memory 1004 (asopposed to storage 1006 or elsewhere) and operates only on data in oneor more internal registers or internal caches or in memory 1004 (asopposed to storage 1006 or elsewhere). One or more memory buses (whichmay each include an address bus and a data bus) may couple processor1002 to memory 1004. Bus 1012 may include one or more memory buses, asdescribed below. In particular embodiments, one or more memorymanagement units (MMUs) reside between processor 1002 and memory 1004and facilitate accesses to memory 1004 requested by processor 1002. Inparticular embodiments, memory 1004 includes random access memory (RAM).This RAM may be volatile memory, where appropriate. Where appropriate,this RAM may be dynamic RAM (DRAM) or static RAM (SRAM). Moreover, whereappropriate, this RAM may be single-ported or multi-ported RAM. Thisdisclosure contemplates any suitable RAM. Memory 1004 may include one ormore memories 1004, where appropriate. Although this disclosuredescribes and illustrates particular memory, this disclosurecontemplates any suitable memory.

In particular embodiments, storage 1006 includes mass storage for dataor instructions. As an example and not by way of limitation, storage1006 may include a hard disk drive (HDD), a floppy disk drive, flashmemory, an optical disc, a magneto-optical disc, magnetic tape, or aUniversal Serial Bus (USB) drive or a combination of two or more ofthese. Storage 1006 may include removable or non-removable (or fixed)media, where appropriate. Storage 1006 may be internal or external tocomputer system 1000, where appropriate. In particular embodiments,storage 1006 is non-volatile, solid-state memory. In particularembodiments, storage 1006 includes read-only memory (ROM). Whereappropriate, this ROM may be mask-programmed ROM, programmable ROM(PROM), erasable PROM (EPROM), electrically erasable PROM (EEPROM),electrically alterable ROM (EAROM), or flash memory or a combination oftwo or more of these. This disclosure contemplates mass storage 1006taking any suitable physical form. Storage 1006 may include one or morestorage control units facilitating communication between processor 1002and storage 1006, where appropriate. Where appropriate, storage 1006 mayinclude one or more storages 1006. Although this disclosure describesand illustrates particular storage, this disclosure contemplates anysuitable storage.

In particular embodiments, I/O interface 1008 includes hardware,software, or both, providing one or more interfaces for communicationbetween computer system 1000 and one or more I/O devices. Computersystem 1000 may include one or more of these I/O devices, whereappropriate. One or more of these I/O devices may enable communicationbetween a person and computer system 1000. As an example and not by wayof limitation, an I/O device may include a keyboard, keypad, microphone,monitor, mouse, printer, scanner, speaker, still camera, stylus, tablet,touch screen, trackball, video camera, another suitable I/O device or acombination of two or more of these. An I/O device may include one ormore sensors. This disclosure contemplates any suitable I/O devices andany suitable I/O interfaces 1008 for them. Where appropriate, I/Ointerface 1008 may include one or more device or software driversenabling processor 1002 to drive one or more of these I/O devices. I/Ointerface 1008 may include one or more I/O interfaces 1008, whereappropriate. Although this disclosure describes and illustrates aparticular I/O interface, this disclosure contemplates any suitable I/Ointerface.

In particular embodiments, communication interface 1010 includeshardware, software, or both providing one or more interfaces forcommunication (such as, for example, packet-based communication) betweencomputer system 1000 and one or more other computer systems 1000 or oneor more networks. As an example and not by way of limitation,communication interface 1010 may include a network interface controller(NIC) or network adapter for communicating with an Ethernet or otherwire-based network or a wireless NIC (WNIC) or wireless adapter forcommunicating with a wireless network, such as a WI-FI network. Thisdisclosure contemplates any suitable network and any suitablecommunication interface 1010 for it. As an example and not by way oflimitation, computer system 1000 may communicate with an ad hoc network,a personal area network (PAN), a local area network (LAN), a wide areanetwork (WAN), a metropolitan area network (MAN), or one or moreportions of the Internet or a combination of two or more of these. Oneor more portions of one or more of these networks may be wired orwireless. As an example, computer system 1000 may communicate with awireless PAN (WPAN) (such as, for example, a BLUETOOTH WPAN), a WI-FInetwork, a WI-MAX network, a cellular telephone network (such as, forexample, a Global System for Mobile Communications (GSM) network), orother suitable wireless network or a combination of two or more ofthese. Computer system 1000 may include any suitable communicationinterface 1010 for any of these networks, where appropriate.Communication interface 1010 may include one or more communicationinterfaces 1010, where appropriate. Although this disclosure describesand illustrates a particular communication interface, this disclosurecontemplates any suitable communication interface.

In particular embodiments, bus 1012 includes hardware, software, or bothcoupling components of computer system 1000 to each other. As an exampleand not by way of limitation, bus 1012 may include an AcceleratedGraphics Port (AGP) or other graphics bus, an Enhanced Industry StandardArchitecture (EISA) bus, a front-side bus (FSB), a HYPERTRANSPORT (HT)interconnect, an Industry Standard Architecture (ISA) bus, an INFINIBANDinterconnect, a low-pin-count (LPC) bus, a memory bus, a Micro ChannelArchitecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, aPCI-Express (PCIe) bus, a serial advanced technology attachment (SATA)bus, a Video Electronics Standards Association local (VLB) bus, oranother suitable bus or a combination of two or more of these. Bus 1012may include one or more buses 1012, where appropriate. Although thisdisclosure describes and illustrates a particular bus, this disclosurecontemplates any suitable bus or interconnect.

Herein, a computer-readable non-transitory storage medium or media mayinclude one or more semiconductor-based or other integrated circuits(ICs) (such, as for example, field-programmable gate arrays (FPGAs) orapplication-specific ICs (ASICs)), hard disk drives (HDDs), hybrid harddrives (HHDs), optical discs, optical disc drives (ODDs),magneto-optical discs, magneto-optical drives, floppy diskettes, floppydisk drives (FDDs), magnetic tapes, solid-state drives (SSDs),RAM-drives, SECURE DIGITAL cards or drives, any other suitablecomputer-readable non-transitory storage media, or any suitablecombination of two or more of these, where appropriate. Acomputer-readable non-transitory storage medium may be volatile,non-volatile, or a combination of volatile and non-volatile, whereappropriate.

Herein, “or” is inclusive and not exclusive, unless expressly indicatedotherwise or indicated otherwise by context. Therefore, herein, “A or B”means “A, B, or both,” unless expressly indicated otherwise or indicatedotherwise by context. Moreover, “and” is both joint and several, unlessexpressly indicated otherwise or indicated otherwise by context.Therefore, herein, “A and B” means “A and B, jointly or severally,”unless expressly indicated otherwise or indicated otherwise by context.

The scope of this disclosure encompasses all changes, substitutions,variations, alterations, and modifications to the example embodimentsdescribed or illustrated herein that a person having ordinary skill inthe art would comprehend. The scope of this disclosure is not limited tothe example embodiments described or illustrated herein. Moreover,although this disclosure describes and illustrates respectiveembodiments herein as including particular components, elements,feature, functions, operations, or steps, any of these embodiments mayinclude any combination or permutation of any of the components,elements, features, functions, operations, or steps described orillustrated anywhere herein that a person having ordinary skill in theart would comprehend. Furthermore, reference in the appended claims toan apparatus or system or a component of an apparatus or system beingadapted to, arranged to, capable of, configured to, enabled to, operableto, or operative to perform a particular function encompasses thatapparatus, system, component, whether or not it or that particularfunction is activated, turned on, or unlocked, as long as thatapparatus, system, or component is so adapted, arranged, capable,configured, enabled, operable, or operative. Additionally, although thisdisclosure describes or illustrates particular embodiments as providingparticular advantages, particular embodiments may provide none, some, orall of these advantages.

What is claimed is:
 1. A method, by a computing system, comprising:accessing an image to be displayed by a display; determining one or morefirst characteristics associated with a content of the image, whereinthe one or more first characteristics comprise a contrast level of thecontent of the image with respect to a background of the image;determining a first display persistence time period for the display todisplay the image based on the one or more first characteristicsassociated with the content of the image; and configuring the display todisplay the image using the first display persistence time period. 2.The method of claim 1, wherein the contrast level is determined based onfirst pixel values of the content of the image and second pixels valuesassociated with the background of the image, and wherein the backgroundis associated with a portion of the image.
 3. The method of claim 1,wherein the contrast level is determined based on first pixel values ofthe content of the image and a brightness level of the background of theimage as measured by one or more sensors, and wherein the background ofthe image is associated with a real world scene.
 4. The method of claim1, further comprising: determining, using a prediction model, alikelihood level of the image to have visible motion blur when displayedby the display.
 5. The method of claim 4, wherein the prediction modelcomprises a mapping model that maps the one or more firstcharacteristics to the first display persistence time period based onthe predicted likelihood of the image to have the visible motion blurwhen displayed by the display.
 6. The method of claim 5, wherein themapping model comprises a lookup table or a transfer function that mapsthe one or more first characteristics to the first display persistencetime period.
 7. The method of claim 1, wherein the content of the imageis associated with a foveated region encompassing a gazing point of auser, and wherein a portion of the image falling outside the foveatedregion is excluded from determining the one or more firstcharacteristics associated with the content of the image.
 8. The methodof claim 1, wherein the first display persistence time period for theimage comprises an absolute time period value or a display duty cycle inpercentage.
 9. The method of claim 1, further comprising: determining,using a head tracking system, a head motion velocity of a user;determining a second display persistence time period for the image basedon the head motion velocity of the user; determining an optimizeddisplay persistence time period based on the first display persistencetime period and the second display persistence time period; andconfiguring the display to display the image using the optimized displaypersistence time period.
 10. The method of claim 9, wherein theoptimized display persistence time period is a shorter displaypersistence time period selected from the first display persistence timeperiod and the second display persistence time period.
 11. The method ofclaim 9, wherein the optimized display persistence time period is aweighted average of the first display persistence time period and thesecond display persistence time period.
 13. The method of claim 9,further comprising: determining a third display persistence time periodfor the image based on a spatial frequency of the content of the image,wherein the optimized display persistence time is determined based onthe first display persistence time period, the second persistencedisplay time period, and the third display persistence time period. 14.The method of claim 13, wherein the optimized display persistence timeperiod is a weighted average of the first display persistence timeperiod, the second display persistence time period, and the thirddisplay persistence time period.
 15. The method of claim 13, wherein theoptimized display persistence time period is a shortest displaypersistence time period selected from the first display persistence timeperiod, the second display persistence time period, and the thirddisplay persistence time period.
 16. The method of claim 1, furthercomprising: determining one or more second characteristics associatedwith the content of the image, wherein the one or more secondcharacteristics comprise a temporal frequency in a temporal frequencydomain; determining a frame rate for the image based on the one or moresecond characteristics associated with the content of the image, whereinthe display is configured to display the image using the frame rate. 17.The method of claim 16, wherein the frame rate is determined based onthe first one or more first characteristics comprising the spatialfrequency in the spatial frequency domain and the one or more secondcharacteristics comprising the temporal frequency in the temporalfrequency domain.
 18. The method of claim 1, wherein the first displaypersistence time period is determined based on a time period thresholdto avoid flicker artifacts, and wherein the first display persistencetime period is longer than the time period threshold.
 19. One or morecomputer-readable non-transitory storage media embodying software thatis operable when executed to: access an image to be displayed by adisplay; determine one or more first characteristics associated with acontent of the image, wherein the one or more first characteristicscomprise a contrast level of the content of the image with respect to abackground of the image; determine a first display persistence timeperiod for the display to display the image based on the one or morefirst characteristics associated with the content of the image; andconfigure the display to display the image using the first displaypersistence time period.
 20. A system comprising: one or moreprocessors; and one or more computer-readable non-transitory storagemedia coupled to one or more of the processors and comprisinginstructions operable when executed by one or more of the processors tocause the system to: access an image to be displayed by a display;determine one or more first characteristics associated with a content ofthe image, wherein the one or more first characteristics comprise acontrast level of the content of the image with respect to a backgroundof the image; determine a first display persistence time period for thedisplay to display the image based on the one or more firstcharacteristics associated with the content of the image; and configurethe display to display the image using the first display persistencetime period.